As an important variant of entity alignment (EA), multi-modal entity alignment (MMEA) aims to discover identical entities across different knowledge graphs (KGs) with multiple modalities like images. However, current MMEA algorithms all adopt KG-level modality fusion strategies but ignore modality differences among individual entities, hurting the robustness to potential noise involved in modalities (e.g., unidentifiable images and relations). In this paper we present MEAformer, a multi-modal entity alignment transformer approach for meta modality hybrid, to dynamically predict the mutual correlation coefficients among modalities for instance-level feature fusion. A modal-aware hard entity replay strategy is also proposed for addressing vague entity details. Extensive experimental results show that our model not only achieves SOTA performance on multiple training scenarios including supervised, unsupervised, iterative, and low resource, but also has limited parameters, optimistic speed, and good interpretability. Our code will be available soon.

The massive growth of self-supervised learning (SSL) has been witnessed in language, vision, speech, and audio domains over the past few years. While discrete label prediction is widely adopted for other modalities, the state-of-the-art audio SSL models still employ reconstruction loss for pre-training. Compared with reconstruction loss, semantic-rich discrete label prediction encourages the SSL model to abstract the high-level audio semantics and discard the redundant details as in human perception. However, a semantic-rich acoustic tokenizer for general audio pre-training is usually not straightforward to obtain, due to the continuous property of audio and unavailable phoneme sequences like speech. To tackle this challenge, we propose BEATs, an iterative audio pre-training framework to learn Bidirectional Encoder representation from Audio Transformers, where an acoustic tokenizer and an audio SSL model are optimized by iterations. In the first iteration, we use random projection as the acoustic tokenizer to train an audio SSL model in a mask and label prediction manner. Then, we train an acoustic tokenizer for the next iteration by distilling the semantic knowledge from the pre-trained or fine-tuned audio SSL model. The iteration is repeated with the hope of mutual promotion of the acoustic tokenizer and audio SSL model. The experimental results demonstrate our acoustic tokenizers can generate discrete labels with rich audio semantics and our audio SSL models achieve state-of-the-art results across various audio classification benchmarks, even outperforming previous models that use more training data and model parameters significantly. Specifically, we set a new state-of-the-art mAP 50.6% on AudioSet-2M for audio-only models without using any external data, and 98.1% accuracy on ESC-50. The code and pre-trained models are available at https://aka.ms/beats.

We present a neural flow wavefunction, Gauge-Fermion FlowNet, and use it to simulate 2+1D lattice compact quantum electrodynamics with finite density dynamical fermions. The gauge field is represented by a neural network which parameterizes a discretized flow-based transformation of the amplitude while the fermionic sign structure is represented by a neural net backflow. This approach directly represents the $U(1)$ degree of freedom without any truncation, obeys Guass's law by construction, samples autoregressively avoiding any equilibration time, and variationally simulates Gauge-Fermion systems with sign problems accurately. In this model, we investigate confinement and string breaking phenomena in different fermion density and hopping regimes. We study the phase transition from the charge crystal phase to the vacuum phase at zero density, and observe the phase seperation and the net charge penetration blocking effect under magnetic interaction at finite density. In addition, we investigate a magnetic phase transition due to the competition effect between the kinetic energy of fermions and the magnetic energy of the gauge field. With our method, we further note potential differences on the order of the phase transitions between a continuous $U(1)$ system and one with finite truncation. Our state-of-the-art neural network approach opens up new possibilities to study different gauge theories coupled to dynamical matter in higher dimensions.

Self-supervised learning (SSL) methods such as WavLM have shown promising speech separation (SS) results in small-scale simulation-based experiments. In this work, we extend the exploration of the SSL-based SS by massively scaling up both the pre-training data (more than 300K hours) and fine-tuning data (10K hours). We also investigate various techniques to efficiently integrate the pre-trained model with the SS network under a limited computation budget, including a low frame rate SSL model training setup and a fine-tuning scheme using only the part of the pre-trained model. Compared with a supervised baseline and the WavLM-based SS model using feature embeddings obtained with the previously released 94K hours trained WavLM, our proposed model obtains 15.9% and 11.2% of relative word error rate (WER) reductions, respectively, for a simulated far-field speech mixture test set. For conversation transcription on real meeting recordings using continuous speech separation, the proposed model achieves 6.8% and 10.6% of relative WER reductions over the purely supervised baseline on AMI and ICSI evaluation sets, respectively, while reducing the computational cost by 38%.

Given sufficient training data on the source domain, cross-domain few-shot learning (CD-FSL) aims at recognizing new classes with a small number of labeled examples on the target domain. The key to addressing CD-FSL is to narrow the domain gap and transferring knowledge of a network trained on the source domain to the target domain. To help knowledge transfer, this paper introduces an intermediate domain generated by mixing images in the source and the target domain. Specifically, to generate the optimal intermediate domain for different target data, we propose a novel target guided dynamic mixup (TGDM) framework that leverages the target data to guide the generation of mixed images via dynamic mixup. The proposed TGDM framework contains a Mixup-3T network for learning classifiers and a dynamic ratio generation network (DRGN) for learning the optimal mix ratio. To better transfer the knowledge, the proposed Mixup-3T network contains three branches with shared parameters for classifying classes in the source domain, target domain, and intermediate domain. To generate the optimal intermediate domain, the DRGN learns to generate an optimal mix ratio according to the performance on auxiliary target data. Then, the whole TGDM framework is trained via bi-level meta-learning so that TGDM can rectify itself to achieve optimal performance on target data. Extensive experimental results on several benchmark datasets verify the effectiveness of our method.

通过探索跨视图一致性，例如，光度计一致性和3D点云的一致性，在自我监督的单眼深度估计（SS-MDE）中取得了显着进步。但是，它们非常容易受到照明差异，遮挡，无纹理区域以及移动对象的影响，使它们不够强大，无法处理各种场景。为了应对这一挑战，我们在本文中研究了两种强大的跨视图一致性。首先，相邻帧之间的空间偏移场是通过通过可变形对齐来从其邻居重建参考框架来获得的，该比对通过深度特征对齐（DFA）损失来对齐时间深度特征。其次，计算每个参考框架及其附近框架的3D点云并转换为体素空间，在其中计算每个体素中的点密度并通过体素密度比对（VDA）损耗对齐。通过这种方式，我们利用了SS-MDE的深度特征空间和3D体素空间的时间连贯性，将“点对点”对齐范式转移到“区域到区域”。与光度一致性损失以及刚性点云对齐损失相比，由于深度特征的强大代表能力以及对上述挑战的素密度的高公差，提出的DFA和VDA损失更加强大。几个户外基准的实验结果表明，我们的方法的表现优于当前最新技术。广泛的消融研究和分析验证了拟议损失的有效性，尤其是在具有挑战性的场景中。代码和型号可在https://github.com/sunnyhelen/rcvc-depth上找到。

本文介绍了一个新型的流媒体自动语音识别（ASR）框架，用于由带有任意几何形状的遥远麦克风阵列捕获的多对话者重叠语音。我们的名为T-Sot-VA的框架在独立开发了两种最近的技术上。基于令牌级别的序列化输出训练（T-SOT），数量几何形状 - 反应连续的语音分离或VARARRARY和流媒体多对话者ASR。为了结合两种技术的最佳，我们新设计了一个基于T-SOT的ASR模型，该模型基于Vararray的两个分离的语音信号生成序列化的多对话者转录。我们还为这种ASR模型提出了一种预训练方案，我们基于单膜单键式ASR训练数据来模拟Vararray的输出信号。使用AMI会议语料库的对话转录实验表明，基于提议的框架的系统大大优于常规的框架。我们的系统分别在保留流媒体推理能力的同时，在多远离微米频道设置中分别实现了AMI开发和评估集的最新单词错误率为13.7％和15.5％。

在复杂的动态环境中，有效的轨迹产生在无人体表面车辆（USV）域中仍然是一个开放的问题。在本文中，提出了针对USV-UAV系统的合作轨迹计划算法，以确保USV可以在多障碍物图中的自主进步过程中执行安全，平稳的路径。具体而言，无人机（UAV）扮演飞行传感器的角色，并提供了实时的全球地图和障碍信息，并具有轻巧的语义细分网络和3D投影转换。然后通过基于图的搜索方法生成初始的避免轨迹。关于USV的独特运动不足的运动学特性，引入了基于船体动态约束的数值优化方法，以使该轨迹易于跟踪进行运动控制。最后，提出了基于在执行过程中具有最低能量消耗限制的NMPC的运动控制方法。实验结果验证了整个系统的有效性，并且生成的轨迹在局部对USV始终具有相当大的跟踪精度。

如今，基础模型已成为人工智能中的基本基础设施之一，铺平了通往通用情报的方式。但是，现实提出了两个紧急挑战：现有的基础模型由英语社区主导；用户通常会获得有限的资源，因此不能总是使用基础模型。为了支持中文社区的发展，我们介绍了一个名为Fengshenbang的开源项目，该项目由认知计算与自然语言研究中心（CCNL）领导。我们的项目具有全面的功能，包括大型预培训模型，用户友好的API，基准，数据集等。我们将所有这些都包装在三个子项目中：风水次模型，风水框架和狂热基准。 Fengshenbang的开源路线图旨在重新评估中国预培训的大型大型模型的开源社区，促使整个中国大型模型社区的发展。我们还希望构建一个以用户为中心的开源生态系统，以允许个人访问所需的模型以匹配其计算资源。此外，我们邀请公司，大学和研究机构与我们合作建立大型开源模型的生态系统。我们希望这个项目将成为中国认知情报的基础。

基于多模式方面的情感分类（MABSC）是一项新兴的分类任务，旨在将给定目标的情感分类，例如具有不同模式的数据中提到的实体。在带有文本和图像的典型多模式数据中，以前的方法不能充分利用图像的细颗粒语义，尤其是与文本的语义结合在一起，并且不完全考虑对细粒图像之间的关系进行建模信息和目标，这导致图像的使用不足和不足以识别细粒度的方面和意见。为了应对这些局限性，我们提出了一个新的框架SEQCSG，包括一种构建顺序跨模式语义图和编码器模型的方法。具体而言，我们从原始图像，图像标题和场景图中提取细粒度的信息，并将它们视为跨模式语义图的元素以及文本的令牌。跨模式语义图表示为具有多模式可见矩阵的序列，指示元素之间的关系。为了有效地利用跨模式语义图，我们建议使用目标提示模板的编码器解码器方法。实验结果表明，我们的方法优于现有方法，并在两个标准数据集MABSC上实现了最新方法。进一步的分析证明了每个组件的有效性，我们的模型可以隐含地学习图像的目标和细粒度信息之间的相关性。